refactor(plugins/compat-oai): use ChatCompletionAccumulator for strea… #3823
Conversation
…ming - Simplified generateStream by using openai-go's ChatCompletionAccumulator - Removed manual tool call accumulation logic (currentToolCall, toolCallCollects) - Created convertChatCompletionToModelResponse helper for unified response conversion - Added support for detailed token usage fields: - ThoughtsTokens (reasoning tokens) - CachedContentTokens (cached tokens) - Audio, prediction tokens in custom field - Added support for refusal messages and system fingerprint metadata - Refactored generateComplete to reuse convertChatCompletionToModelResponse
This commit adds support for configuring whether tools can be executed
concurrently or must be executed sequentially.
Changes:
- Add Concurrent() method to Tool interface to query concurrency support
- Add WithConcurrent() option for configuring tool concurrency settings
- Update all tool definition functions (DefineTool, NewTool, etc.) to
accept optional ToolOption parameters using the functional options pattern
- Store concurrency flag in tool metadata and tool struct
- Modify handleToolRequests() to execute sequential tools first, then
concurrent tools in parallel
- Extract toolExecution struct and executeToolRequest() helper function
to improve code organization and eliminate duplication
By default, all tools support concurrent execution to maintain backward
compatibility. Tools can opt-out by using WithConcurrent(false).
Example usage:
tool := ai.DefineTool(registry, "myTool", "description",
func(ctx *ai.ToolContext, input string) (string, error) {
return "result", nil
},
ai.WithConcurrent(false)) // Sequential execution
|
@hugoaguirre @apascal07 pls review it |
|
Hi Eric, Thank you for this PR. I have given it a quick look over and will do a more thorough review once I investigate how, if at all, the changes to the tools API will interact with our in-progress implementation for multi-part tool responses as you are eager to see supported. We'd like to roll multiple changes to the API into one release. Thanks for your patience! |
|
@apascal07 OK. Thank you very much for your patience in handling this. |
|
Some updates: your idea to support parallel and sequential tools is a good one but we want to take it a step further and not assume that all parallel tools must go first (or vice versa), but rather do stages of all parallel, all sequential, all parallel, etc based on what tool requests the model returns and what the tools are marked with (parallel or not). We're discussing the details of the design now and will implement shortly after. |
2 participants
Description here... Help the reviewer by:
Checklist (if applicable):